Pipeline Setup Via Subscription Manager and SpaceDeck

Setting up a DI pipeline requires the configuration of several components in the following order:

  1. Prepare the Kubernetes cluster

  2. Setup IIDR subscription via the DI Subscription Manager

  3. Create a pipeline using SpaceDeck UI

  1. Prepare the Kubernetes cluster

  2. Create the Space

  3. Create the data source

  4. Create and start the pipeline

Step 1: Prepare the Kubernetes Cluster

For installing a kubernetes cluster, follow the procedure as explained in the Smart DIH Kubernetes Installation page.

Step 2: IIDR Subscription Setup Via DI-Subscription-Manager

If encounter the error "Error creating bean with name 'scopedTarget.embeddedScriptExecutor' defined in URL [jar:file:/home/gsods/app.jar!/BOOT-INF/classes!/com/gigaspaces/di/subscriptionmanager/EmbeddedScriptExecutor.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.gigaspaces.di.subscriptionmanager.EmbeddedScriptExecutor]: Constructor threw exception; nested exception is com.datamirror.ea.api.ApiException: Your account has been locked because the number of consecutive log-in failures exceeded the maximum allowed. Please contact your IBM InfoSphere Data Replication administrator for assistance.", then just execute

kubectl exec svc/iidr-as -- /home/gsods/di-api/unlock_as_user.sh

Port forward DI Subscription Manager: kubectl port-forward svc/di-subscription-manager 6082

Open postman collection via https://elements.getpostman.com/redirect?entityId=22986763-e09fd09c-01da-47f9-892e-66e6783bc547&entityType=collection and use it from your local postman or open the swagger via http://localhost:6082/swagger-ui/index.html#/ and execute following requests:

  1. Execute createDataStore method in Postman or POST /api/v1/datastores in Swagger

  1. Request body:

    {
   "sorName": "ORACLE",
   "dbProvider": "ORACLE",
   "url": "iidr://<IP address of the Oracle server with IIDR Oracle agent>:11001",
   "username": "SYSTEM",
   "password": "<password>",
   "additionalInfo": "string",  
   "offlineMode": true
}

b. Sample output:

    {  
    "name": "ORACLE",
    "hostName": "10.0.0.1",
    "shortVersion": "V11R4M0T4BIIDR_master_5629",
    "database": "JDBC",
    "type": "Dual",
    "status": "",
    "context": ""
}

2. Execute getDataStoreList method in Postman or GET /api/v1/datastores in Swagger

a. Expected output:

    [
     {
        "name": "KAFKA",   
        "hostName": "iidr-kafka@11701",
        "shortVersion": "V11R4M0", 
        "database": "JDBC",
        "type": "Target",
        "status": "Not connected",
        "context": ""
     },
     {  
        "name": "ORACLE",
        "hostName": "iidr-oracle@11001",
        "shortVersion": "V11R4M0",
        "database": "JDBC",
        "type": "Dual",
        "status": "Not connected",
        "context": ""
     }
]

3. Execute createSubscription method in Postman or POST /api/v1/{sourceDataStore}/subscriptions/ in Swagger

a. Parameters:

    sourceDataStore: ORACLE

b. Request body:

    {
   "name": "DEMO"
}

c. Expected output:

    {
   "name": "DEMO"
   "description": "",
   "sourceDataStore": "ORACLE",
   "targetDataStore": "KAFKA",
   "sourceId": "DEMO",
   "tcpHost": "Auto Select",
   "persistency": "Yes",
   "transferableWork": "Target",
   "propagationControl": ""
}

4. Execute getSubscriptionsList method in Postman or GET /api/v1/{sourceDataStore}/subscriptions/ in Swagger

a. Parameters:

    sourceDataStore: ORACLE

b. Expected output:

    [
    {
        "name": "DEMO"
        "state": "INACTIVE",
        "sourceDataStore": "ORACLE",
        "targetDataStore": "KAFKA",
        "latency": "",
        "scheduleEnd": ""
     }
]

5. Execute addTableToSubscription method in Postman or POST /api/v1/{sourceDataStore}/subscriptions/{subscriptionName}/tables in Swagger

a. Parameters:

    sourceDataStore: ORACLE
subscriptionName: DEMO

b. Request body:

    {
   "schema": "RETAIL_DEMO",
   "table": "ORDERS"
}

c. Expected output:

    {
   "sourceTable": "RETAIL_DEMO.ORDERS",
   "target": "Kafka",
   "mappingType": "Avro JSON",
   "method": "Mirror",
   "status": "Refresh",
   "context": "Yes"
}

6. Execute getSubscriptionTableMappings method in Postman or GET /api/v1/{sourceDataStore}/subscriptions/{subscriptionName}/tables in Swagger

a. Parameters:

    sourceDataStore: ORACLE
subscriptionName: DEMO

b. Expected output:

    [
   {
      "sourceTable": "RETAIL_DEMO.ORDERS",
      "target": "Kafka",
      "mappingType": "Avro JSON",
      "method": "Mirror",
      "status": "Refresh",
      "context": ""
   }
]

7. Execute startSubscription method in Postman or POST /api/v1/{sourceDataStore}/subscriptions/{subscriptionName}/start in Swagger

a. Parameters:

    sourceDataStore: ORACLE
subscriptionName: DEMO

b. Request body:

    {}

c. Expected output:

    {
   "subscriptionName": "DEMO",
   "sourceDataStore": "ORACLE",
   "state": "STARTING"
}

8. Execute getSubscriptionStatus method in Postman /api/v1/{sourceDataStore}/subscriptions/{subscriptionName}/status in Swagger

a. Parameters

    sourceDataStore: ORACLE
subscriptionName: DEMO

b. Expected output:

    {
   "subscriptionName": "DEMO",
   "sourceDataStore": "ORACLE",
   "state": "MIRROR_CONTINUOUS"
}

Step 2: Create a Space using SpaceDeck

For creating a Space, follow the procedure as explained in the SpaceDeck – Spaces - Adding a Space page.

Step 3: Create a Data Source using SpaceDeck

For creating a new data source, follow the procedure as explained in the SpaceDeck – Data Sources page, using the following attributes:

  • Data Source Name: ORACLE (name of your choice)

  • Data Source Type: select ORACLE from the dropdown menu

  • URL: iidr://di-oracledb:11001 (iidr://<hostname of oracle agent>:<port>)

  • Username: system

  • Password: admin 11

Step 3: Create a Pipeline using SpaceDeck

Step 4: Create a Pipeline using SpaceDeck

  1. For creating a Pipeline in SpaceDeck, follow the procedure as explained in the SpaceDeck – Data Pipeline – Create New Pipeline page using the following attributes in the Pipeline Configuration screen:

  • Pipeline name: demo 1

  • Space name: demo

  • Connector type: IIDR

  • Data Source Connection: ORACLE

  • CDC Kafka Topic: DEMO

  • Select the RETAIL_DEMO.ORDERS table

2. Create the Pipeline

3. Add a new table to the demo1 Pipeline

  • After clicking Select Tables, select the RETAIL_DEMO schema.

  • Select the CUSTOMERS table

2. Start the Pipeline, select Point in time: EARLIEST

3. Verify that all the data from ORACLE appears in the Space

  1. Add a new row to the ORDERS table (for example, via DBeaver)

  2. Using SpaceDeck Data Query screen (see SpaceDeck – Data Query) type the query "SELECT * FROM ORDERS" and click "Run Query".

  3. Verify that all the data from Oracle is displayed.

3. Verify that all the data from ORACLE appears in the Space

  1. Add a new row to the ORDERS table (for example, via DBeaver)

  2. Using SpaceDeck Data Query screen (see SpaceDeck – Data Query) type the query "SELECT count(*) FROM CUSTOMERS" and click "Run Query".

  3. Verify that all the data from Oracle is displayed.